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This application claims priority of US provisional patent 60/214251 titled "Virtual 
Reality Display Device" filed June 26, 2000 by Angus Duncan Richards. 

The present invention relates generally to various arrangements of optical and electronic 
components to form a high resolution helmet mounted display device (HMD). 

Helmet mounted display devices (HMD's) are rapidly becoming the standard display 
device for virtual reality and "Teleprescence" applications. Such devices generally 
consist of one or more compact image displaying devices mounted on a helmet type 
frame that the viewer wears on their head. The said image displaying devices project 
images into the viewer's eyes via a series of lenses or mirrors so that the viewer perceives 
the image or images to originate from a source outside of the helmet. In the case of 
stereoscopic HMD's a separate image is presented to each of the viewer's eyes so that a 
three dimensional (3D) image can be formed. This 3D image has the additional reality of 
3D depth cues such as stereo parallax (the differential shifting of objects within the 
image due to varying distance from the camera or other imaging source) 

In addition to these depth cues, the viewer's perception that the "virtual" or "synthetic" 
world that is being presented by virtue of the HMD can be further enhanced by 
incorporating a tracking system on the HMD so that as the viewer moves their head 
(pitch, roll or yaw) the projected image moves in a corresponding manner such that it is 
consistent with the formation of images that would have been viewed had the viewer 
been moving in a similar way in the real world. The position of the viewer within the 
virtual world in terms of X,Y,Z (spatial coordinates) is also significant and is often 
utilized to change the projected images to be consistent with the viewer moving through 
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the virtual world. This type of movement is generally not controlled directly by tracking 
systems of the viewer or operator but more generally by virtue of a joystick, data glove, 
cyberpuck or other spatial positioning device. 



At present, the preferred HMD display technology is utilizing compact colour LCD 
display panels. However monochrome 1\CD panels and monochrome and colour CRT 
(cathode ray tube) type displays have also Wen used to good effect. 

The most important factors in the choice of the type of display technology chosen for an 
10 HMD is image quality and the compactness of the display. 



p As HMD's generally have very wide viewiik angles (the angle subtended from the 

^ corners of the image to the pupil of the viewer's eye) the image resolution becomes a 
very important factor. Currently, CRT technologies (particularly monochrome CRT's) 

fe* 15 offer the highest resolutions. Unfortunately they tend to be rather bulky devices and 

§1 \ 

^ require high voltages and have a relatively high power consumption. It is possible to 

* m optically combine several monochrome display devices to form a full colour image. Such 

U \ 

CP techniques are well established and are known to those experienced in the field Utilizing 

m \ 

yp| such techniques it is possible to achieve high resolutions and full colour, however 



^ 20 bulkiness and weight of the resulting display device is often too great for a helmet 
mounted application. 



LCD panels overcome many of these problems py virtue of their compact size and low 
V. power consumption. However they traditionally Irave several disadvantages from CRT's. 
/ 25 The first of these is that their resolution is significWy lower than state of the art CRT's 
/ and their colour saturation is also significantly les^than the corresponding CRT type 
( display. This results in a reduced colour space (as sho^m in fig 37). Traditionally LCD's 

have also had a significantly slower response time (time taken for a pixel to change from 
10% to 90% brightness or vice versa) Which has beema problem for use with rapidly 




changing images, however the newer LCDrechnologies such as TFT and dual active scan 
LCDs have effectively addressed these problems. 

The most significant single problem now\facing designers of HMDs is that of resolution. 
Most current low-medium performance HMD's utilize 0.7" colour LCD's. These offer a 
resolution of approximately 180,000 pixels (red green and blue pixels counted 
separately). This means that the display is\ capable of approximately 60,000 colour 
picture elements. HMD's of this quality display relatively poor images and result in a 
significantly diminished virtual reality (VR) effect. Improved technology in recent years 
is resulting in a new generation of compacA high resolution LCD panels that offer 
significant advantages over these earlier designs*. It is now possible to purchase compact 
colour LCD panels with resolutions as high as\640*480 colour triads (colour picture 
|jj elements). This is equivalent to a pixel count ofj^ approximately 920,000. However at 
€3 present these displays are very expensive resulting in HMD's utilizing these LCD panels 

m \ 

15 to be priced well outside the general computer/gaming market. 

m \ 

Even at these resolutions the viewing angle \s often still reasonably limited which results 
CP in a reduced visual impact of the HMD. Oth&r solutions to this problem which have been 

rJ \ 

■|J} attempted are optical fibre display's and direct retinal scanning. The first approach 
20 utilizes a bunch of optical fibers to optically couple a high resolution image from a 
relatively bulky remote image source to the HMD. This makes possible a display with a 
colour pixel count in the 1,000,000 region Whilst retaining a lightweight HMD. 
Unfortunately this type of display tends to be extremely expensive and although 
technically feasible is priced way out of the general computer/gaming market. The 
second approach is very new and at the present^time would seem to be still at the 
theoretical/early prototype stage. The basic approach is to scan a coloured beam of light 
(probably consisting of three concentric beams of red! green and blue light) directly onto 
the retina of the viewer's eye thereby rendering an image. The approach seems to have 
many merits. The first and foremost being that it is theoretically possible to achieve high 
30 resolutions. In addition, if micro LASERs or LEDs are used as the light source then (by 
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virtue of their high colour purity) a much enhanced colour space is achievable. 
Theoretically the colour pace of such a display could be significantly superior to that of a 
CRT type display. Although theoretically attractive, this approach has several major 
technical hurdles that have to be overcome for the device to become a commercial 
success. It would appear that, at this stage these technical hurdles are the generation of 
sufficiently compact light sources that can proauce a collimated full colour scanable light 
beam and the generation of a very compact high speed optical scanning apparatus for 
scanning the said beam of light across the viewens retina. 
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*g An alternative approach that I have developed is a "mid-ground" between the two 

approaches. It is possible to utilize a Micro-Mirror-Device such as those produced by 
fU \ 

§# 15 Texas Instruments to direct monochrome light\from a compact source through a series 

01 \ 

of lenses, mirrors, or a combination of both, to the eyes of a viewer to produce a high 

^ resolution colour image. Further, it is possible by changing the colour of the monochrome 

light and by projecting the image as a sequence of the red green and blue (or other 

acceptable optical primaries such as orange, green and blue) components of the image to 

20 produce a full colour image with a colour resolution equivalent to the resolution of the 

digital micro-mirror device (DMD) i.e. the apparent pixel resolution will be three times 

as great as the actual resolution of the DMD chip.VThis configuration has all of the 

advantages of both of the previous HMD display technologies. Firstly the optical system 

is relatively simple and requires none of the complex active components that comprise 

the scanning device of the direct retinal display. Howeve^ the DMD technology can offer 

significantly higher resolutions than the LCD technology. DMD chips have been 

fabricated with resolutions close to the 2,000,000 pixel n^ark, which would result in an 

equivalent HMD resolution of 6,000,000 pixels per eye^ The DMD chips are very 

compact and readily lend themselves to incorporation into a^HMD shell. In addition, the 

30 purity of the light source bounced off the DMD is entirely independent of the DMD chip. 




Thus it is possible by using LED or LASI21 light sources to achieve an optimal colour 
space that could easily surpass CRT type dismays (see fig 37). 

There are many reasons why this design is the preferred implementation.. Firstly it offers 
the best features of all previous HMD technologies, secondly it is proven technology and 
thirdly by virtue of the DMD fabrication techniques, it lends itself towards the VR 
environment better than any other display technology. An example of this is shown in fig 
35. As can be seen, it is possible to produce several DMD arrays on a single chip. 
Ordinarily this configuration would be unacceptable for video projection as it would 
10 produce dark lines at the interstices of the DMD* arrays. However in the case of VR the 
surrounding display regions fall into the peripheral vision part of the viewer's eye's and 
thus the viewer is relatively insensitive to the image\discontinuity. 





U 15 THE DIGITAL MICRO-MIRROR ARRAY: 



A micro-mirror array consists of a large numbefi of tiny square mirrors which are pivoted 
at the corners as shown in fig 1. The mirrors can be tilted approximately +-10 degrees by 
applying the appropriate control signals to the digital micro-mirror (DMM) device. The 
angle of tilt of the mirror is digital in nature (hence the name digital micro mirror device) 
that is to say, the mirror can be either in the +10 ofi -10 degree tilt but not in any position 
in between. Analogue control of the light level of tne red, green and blue components for 
each pixel is achieved by varying the time in which the mirror stays in each of the two 
states (i.e. angle of tilt). One of the states (phase 1) is the active state, i.e. when the mirror 
is in this state, light will be reflected into the viewing optics of the HMD. The other state 
(phase 2) is the rest state in which light is reflected a^vay from the viewing optics. By 
altering the proportion of time that each mirror in the DMM stays in one of the two states 
it is possible to simulate any analogue level from zero to\ 100 percent. It should be noted 
however that the break down of time allocated to each of the primary colours does not 
30 necessarily follow that defined in the colour standards This is because the time 



allocation is affected by the relative brightness of the component light sources used to 
illuminate the DMD. For example, if the\red light source was proportionately brighter 
than the green and blue light sources as mm be the case for an incandescent light source 
then the colour can be corrected by simmy reducing the proportion of time that is 
5 allocated to displaying the red component of the image. This characteristic is of great 
benefit to the designer because it makes possible an optimization of the display to the 
available light source. This is particularly important if LEDs are to be used as the light 

source as the blue LEDs tend to have a lower intensity than their red and green 

1 

counterparts. Their are of course limits to how far the time distribution can be skewed. 
10 These limits are defined mainly by the traverse^rate or response time of the DMM array. 

At present each mirror can respond in approximately 20 microseconds. During this period 
q of transition the mirror is an indeterminate state and may or may not contribute to the 
|jj image brightness of the given pixel. During thislperiod the mirror can be considered as 
# providing optical noise (uncertainty) to the image! Given that there are two transitions per 

jU 15 colour period. The colour resolution (number of quantizing levels) for each colour can be 

JP 1 

determined from the amount of time available for each colour display "field" and the 

^ response time of the DMD. Obviously if each colo^ir is to be allocated the same number 

of quantizing levels there should be an equal time allocated to each of the primary 

colours. In such a case the total time for the ima^e "frame" would comprise of three 

20 equal "colour field" periods. This optimal situation is not achievable if there is also a 

delay in changing the light source from one colour jto the next, as is the case in DMM 

projectors utilizing colour wheels. In this case there is also a "dead period" in which the 




DMM array sends all the mirrors to the phase 2 sfate until the colour transition has 
completed. This "dead period" can*be eliminated by utilizing light sources such as LEDs 
25 or LASERS which have very fast switching times. * 

The primary advantage that DMM technology has over LCD and CRT technologies is 
that there is a total absence persistence of image, u.e. as soon as the time period for each 
colour ends there is no trace of the previous image. This characteristic makes possible the 
integration of the three monochrome images into attull colour image. In addition, this 
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characteristic also makes possible the generation of two separate images from a single 
DMD. The generation of multiple images! from a single DMD is possible without 
degrading the spatial resolution of the images. However as there is only a finite 
"window of time" available to complete the six colour images (left image red, green and 

5 blue then Right red, green and blue) the multiplexing of the two images (left and right) 
will result in either a reduction in the refresh rate of the display or a reduction in the 
colour resolution. With the present DMM technology, the multiplexing of two images 
from a single DMD will result in reducing the colour resolution from 8 bits per colour 
(24 bits total) to 7 bits per colour (21 bits total) ona reduction of the refresh rate from 60 

10 frames per second to 30 frames per second). Of theltwo alternatives, a reduction in colour 
resolution is probably the preferred choice as a 30 frame per second image flicker is 
perceivable by the human eye. This drawback istonly a temporary consideration as 
improvements in the response speed of the DM^I devices will widen the colour 
resolution accordingly. 
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As the DMD chips are relatively expensive, reducing the requirement from two devices 
per HMD to a single device offers great advantages to the manufacturer. Several single 
chip DMD HMD designs are shown in figs 7- 14 and figs 17-24. As can be seen in fig 4, 
by virtue of the diagonal hinge arrangement of the DMD the reflected image follows a 
line at substantially 45 degrees to the base of the DMD chip. As a result, the two image 
phases are vertically as well as laterally displaced. This can be advantageous because if 
two separate light sources (each containing the three p^mary colours) are positioned such 
that they are in the same vertical plane but displaced above and below the optical axis of 
the DMD as shown in figs 5-6 then the resultant reflected images from the DMD will be 
25 such that a separate image from each light source will^result in the same horizontal 
optical plane with its counterpart image being substantially displaced from the said 
horizontal optical plane. It then becomes possible by the^orrect synchronization and 
switching of the two light sources to produce two separate images that pass through 
separate optical components to the viewer's eyes. The resultant images will then be 
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perceived by the viewer as originating at some distance outside of the HMD. Several 
different configurations of this design are shown in figs 7-14 and figs 17-24 



5 MIRROR DESIGNS: 

The basic design of a mirror HMD relies hn a light source bouncing light directly off the 
DMD. The reflected light then strikes la concave mirror either directly or by first 
bouncing off one or more plain mirrors. \The concave mirror can be either totally or 
partially silvered. If a partially silvered mirror is utilized it is possible to overlay the 
virtual graphics over the real world. This technique is not possible with a simple lens 
based system because the two images have different focal distances. In all cases the 
final image is viewed directly from the concave mirror. These designs are outlined in 
figs 7-14. Most of the illustrations show vanations designed to make the HMD more 
15 compact. This is an important consideration because the horizontal deviation angle of the 
two reflected images (phase 1A, phase 2B) \as shown in fig 5 is relatively small 
(approximately 14 degrees). As a result the required distance for the two reflected 
images (phase 1A, phase 2B) to separate the required 65 degrees (average inter-ocular 
displacement) is relatively long. As can be seen in\ fig 9 the introduction of an additional 
20 plane mirror/mirrors reduces this distance to half. \The enhancements shown in figs 11- 
12 are aimed at reducing the obstruction to the viewer eyes and removing a spatial 
conflict between the light sources and the DMD. The optimal embodiment of the mirror 
HMD is shown in fig 14. The utilization of thel partially silvered mirror in the 



configuration shown in fig 14 makes the HMD more compact and makes possible the use 
25 of larger light sources which in turn enlarge the size of the region within which the 
viewer can effectively see a clear image (i.e. larger interlocular acceptance region). 

The 2-stage dual mirror hybrid HMD shown imfig 17 is a combination of lens and 
mirror technologies. This design overcomes a size requirement problem of the 
30 secondary mirror that is encountered with the two sWe mirror designs of figs 9-14. 
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However the introduction of the second set of curved mirrors may introduce additional 
optical distortion. } 



5 DESIGN CONSIDERATIONS: 

The minor design HMDs have other advantages in addition to reducing the number of 
required DMD chips to one per HMD. ThA first is that mirrors do not suffer from 
chromatic aberration as do single element lens&s and although more complex lenses such 
as achromatic doublets can significantly overcame these problems they then become at 
least moderately expensive items that add to the cost of the overall system. In addition, it 
q has been found that the proximity of the lens to the viewer's eye causes significant 
|jj problems such as fogging of the optics (due to moisture from the viewer's eyes and face). 
$ This problem has not been found to be significant when mirrors are used, due to the 

fU \ 

|4 15 greater distance between the viewer's eyes and the mirror surface. This greater proximity 
may also have significant safety implications to the eye's of the viewer if the HMDs are 
to be used in environments where there is a risk of tl\e operator falling or the occurrence 
of a collision between the HMD and another object. 

Although optimal in many regards, the mirror designs do have the disadvantage that they 
tend to be more bulky than a more conventionaluens based optical design. In addition it 
is more difficult to attain a wide viewing angle with mirror based designs than it is with 
the lens based designs. Then at the sacrifice of thefce problems I have outlined the basic 
design of several configurations of lens based HMDsWhese are shown in figs 15,16 and 
fl5 figs 18-24. The designs shown in Figs 15 and 16 require two DMD chips per HMD 
however those shown in figs 18-24 produces two separate images from a single DMD as 
in the mirror designs. The dual DMD designs tend to before compact but image quality 
and viewing angle are similar for both designs. \ 



9 



The hybrid HMD designs as shown in figs 15l 17 and fig/s 20-24 utilize a focusing lens 
(or lens combination) positioned directly in front of the DMD to bring the reflected light 
from the DMD to a focus point. This has the advantage of reducing the required size of 
the secondary mirror (in the case of the design Shown in fig 17) or focusing lens (in the 
case of the design shown in figs 21-23). In the case of the design shown in fig. 15 these 
focusing optics serve to reduce the size requirement of the eyepiece. In the case of the 
design shown in fig. 19 this focusing lens has been replaced by a concave mirror. 
Although this has the advantage of removing any possibility of chromatic aberration it 
produces a configuration that is not as compact as tha\^tilizing lenses. 

One of the disadvantages of a lens based design is the requirement for inter-ocular 
adjustment. However as shown in fig 18 and figs 21-22 this can be readily implemented 
by adjusting the spacing between the off-axis focusing lenses in a plane substantially 
perpendicular to the optical axis of the focused light. It should be noted that the focusing 
lenses have a preferred design which has the optical axis of the lens shifted towards the 
inner edge of the lens as shown in fig 18 and figs 21-22. This has the added effect of 
altering the center line of the focused light from a\diverging angle to a substantially 
parallel line. This is a requirement to reduce eye strain. I.e. ideally the viewer should 
perceive the image as originating from some distance in front of the HMD, in which the 
light rays from a distant object will be substantially parallel upon entering the viewer's 
eyes. 

The lens HMD enhancements shown in figs 2B-24 are concerned with removing the 
viewing obstruction produced by the DMD. Although in later designs this may become 
a serious consideration, the present DMDs are sufficiently compact to produce a practical 
HMD utilizing the design shown in fig 22. 



10 



ILLUMINATION: 

The light source used in conjunction with Ihe DMD is a vital component of the overall 
display system. It must be capable of producing the three primary colours of light (green, 
blue and red or orange) with both high puriw and also have the ability to switch rapidly 
between them. LED and LASER light sources are ideal as they both produce very pure 
light and also have very fast switching times. Of the two LEDs are the preferred option as 
they are more compact and are simpler to drive than LASERs. The only disadvantage 
with the use of LEDs is that the overall Radiant light output is relatively low in 
comparison to other more conventional light sources such as tungsten filament lamps. It 
is possible to utilize such older technologiis (like halogen and krypton lamps) if 
appropriately coloured filters or dichroic filters ^:e placed in the optical path. Such filters 
can be either dynamic (in the form of a colour wheel or cylinder) as shown in figs 25-28 
or static as shown in fig 32 and used in conjunc)ion with other switching elements such 
as high speed LCD shutters (such as ferro-electric shutters) In addition to these 
configurations it is possible to utilize fast response colour LCD panels to act as a 
programmable filter, as shown in fig 30. There are many other potential filter 
configurations that are known to those skilled in^he field. In addition to the use of 
filament type light sources there are several types of gas discharge lamps that can be used 
to good effect. These can have either colour phosphors with the appropriate primary 
colours or can produce white light and be fitted wiL colour filters of the appropriate 
colours. The advantages of these type of lamps is thatuhey have fast response times and 
can be rapidly switched. The disadvantages are that they generally require high voltages 
and are significantly more bulky than a corresponding sc|id state lamp such as a LED. 



It should be noted that although the light sources are depicted in the illustrations as being 
point sources, it is in fact favorable for the sources of light to have a finite size. The 
reason for this design detail is evident if one follows the path of the light rays through the 
optical components to the viewer's eye. In the caseW the mirror design, the curvature of 
the concave mirrors are chosen such that light from the light source (assume a point 
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source for simplicity) will come to a focus at the viewer's eyes. This is a requirement for 
consistency of the image brightness becauseA unlike a diffusely illuminated object, each 
point on the DMD is illuminated by only a sinfele ray of light from the light source. Thus 
for the image to be viewable in its entirety, e4pry ray of light that reflects off the DMD 
surface must enter the eye of the viewer. As thel pupil of the viewer is only in the order of 
5 mm, this is only possible if the light rays all come to a focus at the viewer's eye. The 
disadvantage of this focusing of the light is that if the viewer's eye is not positioned 
exactly at the focus point then the image willlnot be viewable in it's entirety. This 
problem is easily solvable by making the light source of a finite size. This finite size light 
source can be thought of as a cluster of tightly packed point light sources each generating 
a pattern of light rays that come to a focus point in a slightly different location. Thus if 
this "diffuse" light source is used, the entire image Jtill be viewable at a number of points 
which will themselves form a pattern of identical shape as the light source but whose size 
is the inverse of the magnification factor of the mirror^ assembly. I.e. (size of the viewable 
regionalize of light source)*(distance between m^ror and focus point of the light 
source)/(distance between light source and mirror). T^e size of the light sources can be 
increased with a corresponding improvement in the size of the viewable region until the 
resultant reflected images as shown in fig 5 begin to overlap at which point the DMD will 
be unable to switch the incident light correctly. Therefore it should be possible, with the 
correct optical design to generate a HMD which has such a wide viewable region that 
there is no requirement for inter-ocular adjustment (i\e. adjustment for the spacing 
between the eyes). This feature would make the HMD significantly easier to configure 
for an unknown viewer, a particularly important feature for use in the public 



entertainment industry. 
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ENHANCEMENTS: 

In addition to the improved resolution of the DMD over LCD technology, there are also a 
number of optical techniques that can be employed which will improve the apparent 
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resolution of the HMD. The first of these \s to introduce pin-cushion distortion into the 

image. The result of this distortion is to cause the pixels in the center of the image to be 

stacked more closely together than those at me periphery of the image, see fig 36. This 

uneven distribution of resolution mimics the distribution of image sensors in the retina 

5 of the eye. The center region of the eye (fovea) contains the highest density of cones 

(colour image sensors) the surrounding area contains less cones and more rods (B/W 

image sensors) These rods are generally responsible for peripheral vision at a lower 

resolution and the cones are responsible for high resolution colour imaging. Thus it is 

possible, by virtue of this pin-cushion distortion, to improve the apparent overall 

10 resolution of the HMD. This distortion can be\^ntroduced optically by altering the 

curvature of the mirrors or lenses used in the system. It should be noted however that in 

addition to altering the pixel density, this optical! distortion also distorts the image 

significantly. Thus it is necessary to provide a counter distortion (barrel distortion) 

jjp electronically to the images prior to their generation. These distortions can be generated 

H 15 in real-time by digital processing means. The techniques for which are known to those 
ffl \ 
yg skilled in the field. * 

iS 

f. 

0* A second technique that can be employed to improve the apparent image resolution is to 

m \ 

tfi produce several different image fields. A central image field (usually at high resolution) 

if??* \ 

^ 20 and one or more secondary image fields at the satoe or lower resolution. This array of 
multiple fields can be achieved by either compositing the images optically (by using a 
partially silvered mirror or other means or by utilizing an family of DMM arrays as 
shown in fig 35. The incorporation of multiple micro-mirror arrays on a single chip is a 
proven fabrication technique. The only drawback to tl^s technique is that there are small 
25 discontinuities produced between the arrays. This will not be a problem for this 
application as these image discontinuities fall outside \he central image region and are 
thus not particularly noticeable to the viewer. 

It should be noted that wherever concave mirrors are utilized in the HMD design these 
30 can be replaced by a combination of a plain mirror and a Fresnel lens. This combination 
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is functionally identical but is not the preferred embodiment because it introduces 
chromatic aberrations and other Fresnel artifacts to the image (such as shadow lines at 
the border of each lenslet). Additionally, when achromatic doublets are shown in the 
diagrams these are functionally interchangeable with single element lenses, other 
combinations of lenses or Fresnel lenses, although the use of achromatic doublets is the 
preferred implementation. 
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